ae \ ee et 7 *s Sey on paw 3 Pete _ ae Pot eerie, ety ® PRS neg) GUMS BPEL oor ate nae See ‘a ee 2A TAS Ty vo tg NEST ae pete 


a0 . \- _ fe : : _ ; . sod 2 “. + : : feos t e 3 : 4 oe 
é ; : 


ar t : ‘ 


t<- 

a 
” 
Tol 


: 3 DOCUMENT RESUME 
ar 7 


1 8D 187, 760 et ae oe TH 806 267 


; : a \ 4“ eu 
AUTHOR © - ‘Weds'S, David J. 5 too os woe, 7 “s 
" TITLE ‘Computerized Adaptive Performange Evaluation: Final "|. ‘4 
ek. ' Repost, February 1976 through January “1980, ” » a 

» . 2ustizurion " Maunesota Unive, Minneapobis. Dépt. of. Psychology. ° - & 
—~ SPONS. AGENCY --“office of Naval Kesearch, Ariington, Va. Personnel » ov 


a 
“ gt 


and Training Research Programs Office.’ | | oa a 
PUB DATE Feb so =. _ oo ee a 
CONTRACT . N000 t4-70-C+0627 2 * i ' a 
NOTE 29pe 4H _ mon ie 
s ; 4 : \ . ‘ z s . 5 r *, , 
© EDRS PRICE eMDOISPCOL ElyS Postage. + : : > 
‘DESCRIPTORS © “achievewent Gaias; *Achievement Tests; Aptitude 
» . ~ + Tests; *Computer Assisted. Testing; Criterion < ° 
A f 4 ee * . Referenced Tests; Féedback; Infeliectual Disciplines; 
a a \ *Lavent Trait Theory Mastery Tests; Postsecondary 
Cae 2 Te - Gducation; Pretests Posttests, Scoring: Testing 
 *.Poobsens; *lest,Theory; ‘Time Factors (Learning) 7 
TUENTIFI2aS ‘*Adaptive. Testing; Test Length’ . Z 


» 7 y 


‘ :*, = 4 ' , ; ; 
; b 4 Durihg a three-year project (1977219680). on re 
computerized adaptive achievement testing, iten characteristic: curve . 
_ thgozy (ICC) and, adaptive testing: strategies designed almost 
eaQlusivezy for apility testing were applied to achievement testing. 
Adaptive techniques substantialiy ceduced:test Length without ° | 
. * reducing quality, when applied tp ‘three prdblem areas: the ‘yse of 
. | ‘unidimensional item pods, the measurement of multiple content areas, 
‘and mastery testing. The use sof unidimensional Icc in achievement: 
testing. was further supported by the lack of effect on diménsionality 
of ammediate khowledge of resuits. BecauSe some results were 
you inconclusive, more research is.needed on inter-subtest: branching, 
- dimensionality of achievement over time in instruction, and adaptive 
mastery testanygy Winally achievement Measurement lacks a coherent 
frgmework. Norm referenced and criterion referenced testing 
approaches have iittle in common; and they do noe relate to the 
coucial problem of measuring andividual achievement gains during or 
agter instruction. To 'incotporate inter-subtes# branching, inter++time' 
branching, and mastery testing, an adaptive self-referenced testing | 
MOdei 25° proposed. (CP) . _ = == = 2 


oe . 
é | , 
‘ : # oy « 
\ “ : . - 
. , « ‘i: a . 
° , ’ . 
- e 


‘” @BSTRACI 


~~ 


, 


y 


: 
aN 


7 


ao 


i ‘ : ‘ ae . . : 
MEMRAM ae ie ARCO IORI ROI RIOR ICICI RCA BOI AC ie AO a oe Me Kt aK Me oe 6 
i peproductious suppaied oy LDAS are the best that Gan be made. x 
* from we Osagirai document. < a 
ME RR A XR A RB a he at a a, 
. 2 : . 


, . a , . a 


SCOPE OF INTEREST NOTICE 


The ERIC Facility has assigned 
this document tor Processing 
to 


: . 
19 QUI judgement, thi document 
Isalso ol snterest to thd clearing. 
houses nated to the tight, Index- 
ing shou’d reflect ther spactal 
POINTS of viaw, 


. 


rele) 
naa - 


“HLVIVSW SOUNAWLaed 30 SN 


ot. at 
AL NOSE da dee, 


Sag te” 
«UN Uf 
WO8s O3%9I78 sty at. 


Sate 
aN dee 


10 
cos 


“NIDTBO NOM VENTA 


‘ 
ut 


is 
of 
af 
em 
Opes 
Crs 
a: 
oy 2 
Oa, 
nt 
<¢o 
aan 
m3 
om 


m 
2 
Oo 
a 
° 
2 
z 
3 
2 
s 
‘ 


# 

NOILYING? 
40 ALNALISNI TUNOILUN 
3av4a1aM BNOWLY 


“O%G38 NASB Sue 


OTH yeh ay, 
ues re 
BE, 
4. 


“Defense, Advanced 
So tant Research: Dey 


he Office ‘Gt Naval: Researels 4" 
d J. Weiss al “investigator 


Ss 


ERIC 


iin 


; . a 
r is . # + 
oS 4 t ‘. b 
SECURITY CLASSIFICATION OF THIS PAGE (Whe pDete Entered) ; . oe 
READ INSTRUCTIONS 
REPORT NUMGER GOVT ACCESSION NO] 3. RECIPIENT'S CATALOG NUMBER 
: ? 
Dae hoy eeeaee aes Se eee! aan, Le ee ee eT Oe 
4 ITC & fang Sudtitie) : 5. TYPE OF RERORT & PERIOD COVEREO 
: : > Final Report P 
Final Report: Computerized Adaptive February 1976 = January 1980 
Performance Evaluation ; 6. PERFORMING ORG. REPORT NUMBER 
7 AUTHOR a) ° 8 CONTRACT OR GRANT NUMBER(e) 
Navid J. Weiss m & ee, N00014~76-C-0627 . 
ea eae aS Ln Ie El eee ee NS EAC ee ree ee LL ee Se ae eames 
9 PERFORMING ORGAN: ZATION NAME ANC ADORESS 10. PROGRAM a eee NT. PROJECT, TASK 
J AREA 4 W a UNIT NUMBE 
Department of Psychology PE. 61153N PROJ. TRROG2-04 
- University of Minnesota T.A.: RROG2-04-01 
Minneapolis, MN 55455 \ W.U.¢ NRi5SO-389 
“* TONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE _ 
Persunnel and Training Research Programs _Fekruary 1980 __ |. 
Office of Naval Research ‘3 NUMBER OF PAGES .. 4 _s 
MONS TORING AGENCY chat & AODRESS/(it difforent from (Controlting Office; 1S. SECURITY CLASS. (of thie report) 
g : ‘ 
; Unclassified . ; w 
‘e ‘TTSa. DECLASSIFICATION’ DOWNGRADING 
SCHEDULE 


Bann Sesame, rw n<tgemntae eng nt A, a . : — — 2 
16 CIS RI@U TION STATEMENT fof this Report) ‘ 


Approved for public release; distribution unlimited. Reproduction in whole 
er in part is permitted for any purpose of tthe United ehates Government. 
. or 


os i 
17+ 18° RAUTION STATEMENT (of the edetract entered In Black 26, if different trom Report) 


a 


B SUPPLEMENTARY NOTES 
This research was supported by funds from the Defense ‘Advanced ‘Research 
_Projects Agency, Navy Personnel and Research Development Center, Army 
Research Institute, Air Force Human Resources Laboratory, and Office of 
Naval Research, and monitored by the Office of Naval Research. 


f i Eee eee ne err en eee 
19 REY WORDS (Con'inue cn reverse aide if neceesary and identify by block number) 


[esting Sequential Testing | Programmed fostins 

Achievement Testing Branched Testing Response-Contingeéent Testing a 

Coaputerized Testing Individualized Testing Automated Testing » | 

Adaptive Testing ’ Tailored Testing Item Characteristic Curve Theory | 
‘ Latent-Trait Test Theory 


ap ADSTHACT TC onlnun on vavatas aide If Recessaty aad idonilly by blocknumben) = ~SC CCC CCSS::C<“<=i<=~*é‘ 1 _ 
Three vears of research on computerized adaptive achievement -testing are 

summartxed. The-goals of the research program are described, and the research 

fapproach ‘ws summarized and related to the nine technical. reports completed 

funder this contract. Major research findings are presented. The implications | ‘ 

Pot the research metifods and résults for future research in compyterized adap- 

J tive achievement testing are described. Also included are abstiacts of the 

nine technical reports and a list of other project reports and papers. - 


DO 600", 1473 enition oF | Nov 6815 OBSOLETE § iauct eid ; J 4 
S/N 0102-LF-014-66C1 1: beer ttchassitied 
URITY CLASSIFICATION OF THIS PAGE (Wher Data Entered) 


~ 
ae 


w 


Contenta + ? 


t 


poject ives sg ree ena ete eas Nii 


. 4 
Approach shal dar bets seyienssawad se daseqend nent agchadiaeeiaeiayienss NARS 


Oa 


Literatuke Rev tes ck 
oy Revised objectives Se hee PON Pe ee ee eee re eee ee ee 


7 : ’ to “ ‘i 745, 


tae i , 4: * ; X\. 
Results ‘ ss * - ve . ‘ % 
Psu ts eeeeteeeteeeeeeetrteeese#e#wgeeeeee#sseeee e2 eeeeeoee tee 6 ¢€ @ © @ O38 oe oeeeoeeetoeese ee @ 


Applications of Item Characteristi¢ Curve’ Models“and Adaptive 
Testiry Strategies ee ee ee 
ECG MOGGLS 6 6. s6 26466. GR AEE A SO OHS OOO TMAE HS THOOES ESET ORG ER ERE SS 
Adaptive Testing Strategies oo... li ccceccc ence cece enseseeneye 
Tet Seorine MELNOdS <404964-6 054 geaed ane tewGuescbeeeeus eaaens 4 
Unique Problems of Achievement Testing .e.ecseseeeereeceteeceeerecs 
Multiple Content Areas See e meee eee eee teen tense eee rtaseregas 
Mastery Tes tang ee 
Dimensianalftty: of Achievement Over Time ....ccccveccccrwecseees 


Effects of Knowledge of Results eee meee ooo eee sree eee tea ese se Oye oie 


Madar Fiadings PS ASR EE PR Ta EAA VEER RO Rese ey We HEREC ee aS 


tmol teat itis tor hurther, Research a a i ea ee er rie i 


Teter ~Subte st. Branching eo eee er eer er oeo eee reorer eter eoeoereosree esos eeaeoene @ 
Dimensfonality of Achievement Over Time ...ccccsccccecreectereevvees 
y Adaptive Mastery Testing boa OG a Wh wile wie; © BW Beware ORO ALO Orerer OOH eels eee 


\haptive Selt-Referenced Testing Re ee ee rn eer ete ye re are arr a ¥ 


Serie ee - 
ne VT RIVOLI Cw ew he we th wee eee eee HOO HR eee eee eee seer onteoeese ooeveveeevreveoev ee o 8 


2 


= 


Veitrasts of Research Reports cecccecvvcvcceves fag ee sesa gees ox reves stee 
Reaearch Seport 77-5. Calibration of an [tem Pool for the“Adaptive 
Measurement of Achievement UTS Tre ee Skee ee 
Roaearsh Report 77-6. An Adaptive Testing Stratégy for 
+ Se itevemenit Tred BATtEYIOS: 16k side veoo aude o5ee SSA eee ewaas 
fe earel Report 77-7, dAn Information Comparison of Conventional 
* and Adaptive Tests in the Measurement of Achievement ..4.....-- 
Keosearch Report 78%. ‘A Construct Validation of Addptive 
Achievement Testing ccoccecccccrcerereer servers eretsceeeereerens 
hespareh Report 7°-1. Computer Programs for Scoring Test ata ~ 
with Ttem Characteristic Curve Models 2. sce tececac mercer vevece 
Keaearch Report 79-3, Relationships Among Achievement Level , 
Edtimates trom Three Item Characteristic. Curve Scoring Metheds 
Research Report 79*4. Effect. of Point in Time in Instruction on 
the Measurement of Achievement SUG eT eee see ea teu sas 
Research Report 79-5. An Adaptive Testing Strategy for Mastery 
DECTAIGNS “255444 dae ee aewkd $50 Ss Cheese aes bw ewe SA ees REY 
“Research Report 80-1. Effects of Immediate Knowledge of Results 
on Achievement Test aes and Test Dimensional ity Se Gas 


7 ’ 


s 


Ather Project Reports cose cece e eee ee ee eee eee ee eee eee nee eee eee ees 


a 


™Xy, 


2 


~s 


18 


i? 


BWP Wa, Wee 


DADAUUL 


Y, 
4 
a 
‘; 
¢ $ 
LN 
e 
hd 
13 


a 


a 


_ ERIC 


1 
. 
} 
' 


tumbers of items required for adaptive testing methods based on classical {test 


“vriterion-referenced testing, (also known as mastery testing) is quite. different 
“trom that of ability testing. As a result, a serious limitation of the area of 


‘ ‘ : 
e | _ . 
. & ‘ . : y . 
es ; ss ‘ *. 
‘Sy FINAL REPORT: - a | 
— COMPUTERIZED. ADAPTIVE PERFORMANCE EVALUATION 
an a. . i. sdjaeenict aN ‘ ‘ = 


€ U 


The original objectives of this research’ program were concerned with. (1) 


~ 


-. thre develapment of a- psychometric basis for the ccnstruction, development, and. ' e 


evaluation of criterion-referenced performance tests for use in the measure- 
ment of achievement and (2) the dcvelopment of psychometric methodology for 
computerized adaptive performance simulation tests.. A performance simulation 
test. was defined as an interactive problem-solving test in a particular area 
of achievement. ‘ : 


s to . 
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Research in pursuance of these objectives began in February 1976 and con- 


“tinued through January 1979. Technical reports were completed during the pe- 


riod February 1979 through January*1980. ~~ ° 
| AppFoach . 


Literature Review < 


Research began with a review of the literature on the problem of the mea- 
sugement Of performance and achievement. Analysis of the literature concerned . 
with the measurement of achievement led to a restructuring of project objectives. 


Figure 1 summarizes the several approaches to the measuresent of achieve- 
mnttor performance that were identified in the review of the literatyre. As . 
Figure 1 shows, the meaSurement of achievement was determined to be considerably 
more comp Lex than ,the related problem of ability measurement. The most promi- 
nent trend in the ‘achievement measurement literature is the use of population— 
or torm-referenced techniques borrowed from the field of ability measurement. 
lA general, these techniques have been based on classical psychological test 


' theory, with cH® result that the obtained measurements and statements of 


achievement or performance have differed for a given individual based on the 
particuéar norming group to which the individual has been ‘compared. In addi~ 
tian, the use of classical test theory for achievement measurement makes it 
difficult to apply ‘adaptive testing techniques, because of the rezatively large 


theory (Weiss, 1974). ; : : 


e.. 
Thé second major trend ident if ied in the achievement measurement Litera- 
ture wh? that of; Convant= or criterion- -referenced measurement. The problem of 


etiterfton- refetepeed measurement is that the psychometric rationale for it 

was relatively undeveloped. Th addition, virtually no methodologies had been 
deve Loped for the application of adaptive testing techniques to the problem of 
chi tertonpreferenced measurement. « Thus, an important objective ‘of the project 


was f6°Mevise adaptive testing methodologies uniquely applicable to the. prob- : 
lem of criterifon- (or content-) referenced measurement. 
Nw eee 
ie & wd 


ae. i = . - 3 


iy 
A 
a 
Y 
| 
o 
4 
e 
’ 
\ 
“4 
1 
1 
e + 


a i. 6 eae 3 = wy e 4 ae eS. eee 
‘me «+ LONTENT™ O08 CRI TERLON-REFERENCED } “1 [S ; ; StaGe-Rererencen ‘ 
oe 


% . a8 7 . a, ty 
LY s * . 
€ ; : . s % 4 ae 
: : : OR 
' = * 
¥ ‘ > FIGURE 2. S. ) 
. APPROACHES TO ACHIEVEMENT/PERFORMANCE EVALUATION * 


e . ’ 


seeeeen ones = ot ew here: 


a a tg 
Dy 


- * en POETIC 8 Nt FERENCE —__ 
g INDIVIDUALS RELATIVE POSITION TO OTHER INDIVIDUALS 
BASED ON TRADITIONAL PSYCHOMETRIC THEORY ,. 
RESULTS DIFFER YSING mY RFEREN® NORHING GROUPS * 
. , 


a 


t ‘ d * 2 eee 4 m, er 
i REFLECTS PROPORTION OF CORRECT ,! eo Jay REFLECTS WSSIGNMENT TO STAGE \° 
; RESPONSES TO TEST [TEMS = » Appraacnes ACCORUING TO SUPERIMPOSED 
ee | . ¥a \ , | STAGE THEORY “ 
Baged on TEST JrEM POOL, ADEQUATELY 1,4 bm Qi. AN ae: ; 
J REPRESENTING THE DOMAIN OF INSTAUCTIONK ; Ac Levewent/ © SPECIAIP.CASE UF TIME-REFERENCED 
at a ce PERFORMANCE = fF am | LIMITED PSYCHOMETRIC METRODOLOGY | « 
REQUIRES CAREFUL CONCEPTUALIZATION OF | \ _, MEASUREMENT (“ fe 
i ret Vrem Poot \ a P 


1 
} . eae ‘5 ; ; -. | 
INDE VELOPED PSYCHOMETRIC BATRONASE, { ; 7 1 4 L a os a __ 
? 
7 . 


. Peden ege yaeteen ise eases r 
<a 
Le 


} A ° 
ee eee = ae ee See sats 


‘ r Time REFERENCED ~ - , 


, : REFLECTS INDIVIDUAL’S PERFORMANCE AS 
| A FUNCTION OF TIME IN TRAIHING ; 


t | UNDERSCORES NECFSSITY FOR LONGITUDINAL uM \ ; a 
| MEASUREMENT i ' , ° @ 
\ . | TECHNICAL PROBLEM WITH TRANSFORMATION  j : 
; | FROM TEST SCORE TO TIME | , ! B 


| ‘ . 
“ | LITTLE PSYCHOMETRIC RATIONALE AVAILABLE | > : 
cLETDE PSYCHOPE tRIC RATIONAL eer 


bd ‘ 


e . . e . . ran) ri 
The Literature review identified a second important problem uniquely chart 
avteristic of achievement measurement, which was not characteristic of ability 
measurement. This problem was the fact that the @asurement of achievement fre- 
quently occurs as a result of ar individual's expgsure to a‘restricted environ- 
ment, such as a class ‘or a training rourse. Typjcal of these, environments is \ 
a relatively short time-frame in which the chapge in an individual's observed \. 


ichvevement level is to occur. Thus, an impoftani. problem in the area of achieve“ 


ment fleasurement is measuring an, iyedividual's achievement level over relatively 
short periods of time, including;changes in that achievement level as a function 
et ot fine., ; ‘ : . 
é 

_ Such an approach to measurement can be called "time-referen¢ed" measure- 
avnt, which evidences several important problems. Among these are the problem 
ot measuring change in an individual's achievement level from one point in time 
to another relatively close point in time. Similar to the area of criterion- 
referenced measurenent, there was very little psychometric rationale-available’ 
in the Literature for the measurement of individual gain as required by a time- 
referenced measurement perspective. 


A special case of ,time-referenced measurement is that of "stage-referenced" 
measurement. In stage-referenced measurement, a particular theoretical struc- 
ture describing stages of achievement is superimposed on the measurement problem. 
Thus, the achievement measurement problem becomes that of -determining whether 


an individual is progressing in achievement levels according to the particular 


stage theorv deseribing levels of achievement in the specified achievement do- 
main. Similar to the problems of time-referenced’ and criterion-referenced mea~ 
surement of fachievement, there was very little psychometric rationale available 
in the Literature fer ,the stage-referenced measurement of achieveme t. 
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' The review of the literature also jdefftified several other problems that: 
are characteristic of the measvrement of acl\ievement, as compared to the mea- 
surement of ability. One of thes is that the goals of achievement measure- 
ment are frequeritly embodied iy the specification of particular achievement 
domains. - Frequently, these acfevanent domains are relatively specific; and 
in the process of constructing achievement ‘tests to measuré these domains, . . 


only a limited number of test items can be’ generated due to the specificity a 


‘of the domains. Thus,-the measurement of achievement frequently requires a 
multidimengional approach measuring specifi€ content .domains using relatively 
small numbers of test items in comparison to those used for the measurement of 
ability. As a result, traditional adaptive testing models developed in the 
ability testing area may not be directly applicable to'the measurement of 
achievement. The literature thus suggested that it might be necessary to de- 
velop adaptive testing strategies for the measurement of achievement that “were 
specifically designed to operate efficiently with a large number of small’ content 
dSmains. _ “> 
7 ‘Finally, the review of the literature and some subsequent analysis of in- 
structidnal environments indicated that the measurement of performanee by com- 
puterized adaptive simdlation techniques.was considerably more complex’ than 
had originally been anticipated. Additionally, the review indicated that there 
was virtually.no psychometric rationale available in the literature for the 
measurement of perrgrmance by simulation. Although there were some applica- 
ions of performance simulation to the measurement ef achievement, analysis of 
ythe methodologies and attempts to apply those methodologies ingrelevant instruc- 
tional environments ,indicated that the measurement of achieveme by .performance 
simulation was serinusly situation-bound. That is, it was extremely unlikely 
that any generalizable methodologies could be developed that weuld be trans- 
terable across instructional situations of different types. Consequently, after 
some preliminary trial work with performance simulations, the obitective of de- 
veloping a psychometric rationale for the measurement of achievement by pexfor- 
mince simulation was abandoned until more generalizable methodologies could be 
identit £éd. : _ 


Revised Objectives | ee ( 

the review of the Iiterature thus led to a redefinition’ of project goals. 
The revised project objectives were oriented around the development of adaptive 
testing stratezies designed to address the unique problems of the theasurement 
of ashievement. The approach used was to first examine the applicability of 
adaptive testing strategies developed in the ability testing domain to rele- 
vant problems in the achievement testing domain. Then, further efforts were 
orfented toward the development cf adaptive testing techniques specifically 
designed tor the unique demands of achievement testing, and an investigation 
of some of the unique problems of achievement testiag and ‘analysis of some of 
the psvehological aspects of the achievement testing environment. 


Result s 
Applications of Item Characteristic Curve Models and Adaptive Testing Strategies 


‘ . 
[CC inodels. The first technical repott from’the project (Research Report 


77-5) investigated the question of whether item characteristic curve (ICC) the- 
ory methods utilized in ability testing were applicable to data derived from 
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the measurement of achievement. This repprt described ‘the ICC calibration of 
an acht@vement testing item pool., Data ysed were derived from a general bi- : 
ology course at tie University of Minnesgta. The item pool was a multiple- ' 
. choige set of lrems written by course instructors. In addition to analyzing, 
the applicability of ICC item calibration techniques to this-item pool, the 
dimensionality of the poo] was examined in order to determine whether uni- 
diméngional ICC theory was applicable to the measurement .of this: domain of 
: achievement. Results showed that the pool was generally unidimensional and 
that it was possible to derive appropriate ICC parameters’ from this.pool. 
” Adaptive téscing strategies. Using this ‘item pool, the next question in- 
. vestigated was whether adaptive testing techniques developed for the ability 
testing-domain were applicable to the measurement of biology achievement (Re- 
‘search Report 77-7). A stratified-adaptive (stradaptive) test was acministered 
to a group of students and compared with a conventional classroom.test derived 
from the same item pocil, as well as with an improved conventional test devel- 
uped trom the pool. Tests’ were compared in terms of information (precision of 
measurement). Results showed that, as-expected, the adaptive test provided 
meadurement of greater precision than did the conventional tests.’ The results 
alao indicated that the adaptive test provided measurement of equal precision 
with considerably fewer numbers of items than did the conventional tests. 
When the average number of items administered in ‘the adaptive test was equal 
te that of the conventional tests, adaptive test scores were more precise than we 
ef{ther the classroom conventional test or the improved conventional test. 
» / ; ° , 
Although the demonstration of improved precision of measurement from 
tr adaptive testing in comparison to conventional testing is’ supportive of the 
. oweneral value of adaptive testing for measuring achievement, the q¥astion of 
the relative validity of the two techniques was also important. In Research 
Repert 72-4 the comparative validity of adaptive and conventional achievement 
tests Was studied. Since it is very difficult in the achievement domain to ob-~ 
tain a criterion against which the relative validity of two testing techniques 
fan be evaluated, the problem was approached by comparing the respective con- : 
Boruet validity of the two testing techniques. - The results of this study showed 
raat tae canstruet validity of the adaptive tests was effectively higher than 
sat ct the conventional tests, since equal validities were afhieved for the 
yo testine strategies, but the adaptive tests required 25% to 35% fever 
items than did the conventional tests. 


ihus, these studies demonstrated the applicability of ICC techniques pre~ 
viousiy applied almost exclusively in the area of ability testing, ‘as well-as 
ifiptive testing strategies developed for ability testing, to the problem of 
ihievement’ testing. Results indicated both higher precision of measurement 
? and bisher eftective Levels ot validity for the adaptive test. : 


Ice scoring methods. The process of examining the problem of the appli- 
vetbility et ECC theory and adaptive testing techniques to the measurement of 
achievement led to the ‘development of a-set of computer programs for scoring 
Rehievement test data with ICC models. Since these programs were written as, 
veneral purpose programs,.they were made available in Research Repert 79-1 for 
ather pesearchers who desired to use ICC methodologies in scoring achievement 


' "ar abicBty tests. 


In the process of implementing the reliability and validity studies compar~ 
tug adaptive and conventional testing strategies, Jecisions had to‘be made about 
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the appropriate ways ‘of scoring the scWlevenent test data using ICC models., 
These decisions were necessary for both the conventional tests and the adaptive 
tests. Thus, a relevant question concerned the relationships among. achievement 
level estimates using the one-, two-, and three-parameter ICC models, as well 
as the. maximum Likelihood normal, niaximum likelihood -logistic, and Bayesian 

@ methods for scorihg ability test data with ICC models. 


: ; 
To compare these scoring methods and models with each other, live data - 
from an achievement test were scored by all combinations of. ICC models. and 
methods. The results (Research Report 79-3) indicated that highly. similar 
achievement -level estimates were derived from the one- and two-parameter data ~ 
but that when the third (guessing) parameter was added to the scoring proce- 
dures, the similarities, among achievement level estimates decreased. The data. 
also indicated that the’ three-parameter model resulted in less similar achiéve- 
ment level estimates for adaptive test data than for conventicnal test data. 
However, at the same time, there were fewer convergence failures for maximum . 
lLikelLhood scoring in adaptive test data than there were in conventional test 
data. a P a : 


tnique Problems of Achievement Testing 


In addition to studying the applicability of ICC models and adaptive test 
procedures derived from ability testing to the problems of achievement testing, 
the project was concerned with the development of solutions to some of the 
unfijue pronlems raised in achievement testing,as well as the analysis of the 
implications of some other unique characteristics of achievement testing. 


Multiple content areas. As indicated previously, one problem character- 
istic ot achievement ‘testing, in contrast to ability testing, is the necessity 
to measure an individual's achievement levels in a number of content areas at 
the’ same time. In addition, in many cases the number of items available in 
a content area is very restricted, resulting ‘in relatively short tests that 
would not permit thecapplication of many standard adaptive testing strategies. 


’ 
1 . 


| ‘ 
Consequontlyv, an adaptive testing strategy designed specifically for 


achievement test batteries was developed (Research Report 77-6). This strategy 

iy one that is applieable.to achievement tests composed of any number of short 
subtests. The stcategv,is designed to utilize both intra-subtest adaptive item 
selection as well as inter-subtest adaptive branching in order to reduce test bat- 
tery iength to a minimum for each individual. The testing strategy utilizes a 
maximum information [CC-based f'tem selection technique combined with Bayestan 
seoring to adaptively select items within a subtest until there are no items 

left that provide more than trivial amounts of information about an individual's 


achievement level, Having obtained an achievement level estimate from one sub- 
. - test, that estimate is then used in a hivariate regression equation to obtain 

a prior achievement level estimate in the next subtest in the test Dattery. 
‘ The adaptive testing strategy then adaptively selects items iu the next subtest, 


using the prior ability estimate, until no further items are available for ad- 
ministration in that subtest. At the end of the second subtest, multiple re- 
gression is used to obtain a prior achievement level estimate to“begin testing 
in the third subgest, and the process is repented until all subtests have been 
administered. 
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Results “of applying this adaptive testing. strategy to an achievement’ test 
\batrery in a military testing eravironment, using real-data, simulation tech- | 
niques, indicated an average 504 reduction in test length for the individuals 
tested, with no loss in the quality of the obtained measurements. Test length- 
reductions var ted’ from 18% to 80% across individuals. Thus, considerable re- 
ductions in number of test items administered wera achieved while maintaining 
the quality of the measurements obtained from the convent ional test... . ‘ 


Mastery testing. Since thr gb hoadliogie. required to Saapeively measure 
mastery within a criterion- referenced framework are rvi the same as those avail- 
able for the measurement of ability levels, an adaptive testing strategy for 
making mastery decisions was developed (Research Report 79-5). This testing 
stratepy utilized ICC theory and methodologies in conjunction with a maximum 
information adaptive testing technique and Bayesian scoring. The testing strat- 
egy was designed to use a prespecified and flexible mastery level for comparison 
with each individual" S$ 3 performance. ° 


s 
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The adaptive mastery Lescing strategy was compared with a sorvedelonel 


*mastery test in a military tralining environment, using real--data simulation. 


When fthe results‘tfor the two-testing strategies were compared, the adaptive’ 
mastery Peer ine strategy reduced® the average test ‘length from 30% to 81% over 

all mastery decisions examined! with modal test length reductions up to 922, 

vet it reached the same decision as the conventional tést for 96% of trainees. ~- 
Thus, again, cowsiderable savings’ in the number of test items adminigtered were. | 
observed for the adaptive test, while it made decisions which were highly similar 
to those made by the conventional test. i 


. 


Dimensionality ¢ of achievenient over time. As indicated above, a unique 
problem in the area of the measurement of achievement is that of measuring a 
person's change in achievement level ‘over a relatively short period of time. 
It ICC theory is to be used in the measurement of achievement, it will gain 
its highest degree of potenti at usefulness if it can be used to measure the ~ 
wrowta’ in one individual! s achievement Mvel from the beginning of iastruction 
to later points in instruction. However, the implementation of this paradigm 
for the measurement of individual growth requires the demonstrat ion that an 
achievement test given at two or more ‘points in time feasures the same achieve- 
ment dimension and that the dimension measured is a undimensional variable. 
Research Report 79-4 reported results addressed to this question.. 
a 

Dimensfonality was investigated within the pretest-test paradigm for mea~ 
suring change in achfevement levels and within the test-posttest paradigm, for 
measuring retention, cata indicated that there were some questions about the 
uttlity of the pretest-test paradigm, since’ a comparisan of the ICC parameter 
estimates obtained from achievement test items at two points in time 4 weeks 
apart suggested a change in the dimensionaLity of achievement over that period 
of anstruction. These results were also supported by the results.of factor 
an ses. The’Hata did, however, support the test-posttest paradigm to measure 
retention, since a regression comparison of students' achievement level estimates 
did not indicate any differences in the achievement metric up to 1 month after 
the completion of instruction. ‘However, additional résearch is necessary in 
order to further verify and examine these conclusions. , 


Effects. of knowledge of results. The advent of computerized adaptive 


“testing ‘also . brings with it the- -potent ial of aia: to studente during 
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the process of testing immedigte feedback as to the correetness or incorrect- : 
ness of their tést responses, Previdus research in the ability testing domain . 
istration of immediate knowledge of résulté for each test item during the 
process of testing reduces the effects of extraneous variables on ability test 
scores. However; if immediate feedback is to be administered to students in: 

an achievement testing environment, it ‘is possible that the information gain- 


‘ed from feedback on prior items‘may affect ‘a ‘student ' s performance on subse- 
“quent items in the test. A babic assumption of ICC theory is that of local 


independence, that is, that the response of a student to a‘given test item is 
the result: qnly of the underlying achievement variable, and not of’ other 


‘variables. If knowledge of results from prior itéms in an achievement test 


affected a student's performance on subsequent items, the assumption of local 
independence would be violated.’ 

Reasarch Report 80-1 was concerned with this issue. In two studies, . 
data derived from two groups of' students (one of which received immediate 
knowledge of results while the other received no knowledge of je resuge on 


» 


computer~administered tests were’ compared with each' other. The res th indi- ’ 
cated essentially no systematic differences in!achievement level est mates or 
in the dimensionality of the students' responses as a result of the adminis- 
tration of immediate knowledge of results. Thus,.the data indicated that 
this. added benefit of computerized administration of achievement tests did 
not affect the assumptions under which ICC theory could be applied in the 
achievement testing environment. 
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‘ Major Findings + _ he 


Summarized below are the major findings from this research program, with 
reLerences to the research reports in which these findings are reported. In 
addition to these major. findings, the original research reports should be con- 
sulted fer additional important resulcs and conclusions. 

* 
1, fhe successful application of ICC theory to achievement 
: testing requires that the item! pool be reasonably uni- 
dimensional. Analyses of a large item pool, constructed 
“by the instructional staff of a university level course, 
indicated that the pool was essentially unidimensional 
(Research Reports 77-5 and 80-1). 


+¢ 
-. When ICC item parameters were estimated from this item pool, 
the majority of the items resulted in parameter estimates 
that were suitable for operational testing purposes (Research 
Report 77-5) : 


3. The ICC parameter estimates obtained from this item pool re- 
@/ .  ftrected sufficiently high Levels of discrimination and a 
suffiefent range of difficulty to be useful’ in adaptive 
testing (Resea,'ch Reports 77-5, 77-7,.and 78-4). 


4, Using operational achievement tests from military instruc~ 

tivnal environments, it was possible to obtain usable ICC ; 
item pardmeter estimates even. in narrowly defined content 

domains (Research Reports 77-6 and 79-5). 
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the item parameter data indicate that~some caution might. be 
uecessary, however, when estimating ICC item parameters in 
achievement test data. Relatively high ‘discrimination para~ 
meter estimates in conjunction with high guessing parameter 
estimates (Research Reports 77-5, 77:6, and’ 79-5) may re- 
flect a restriction in range on the achievement variable. 


“rf the efPect of instruction is to eliminate individual 


differences in measured achievement, ICC parameter esti- , 
mates of discrimination and guessing obtained on groups at 


their: peak of tmstruction will-be artificially inflated. 


Additional research on this problem is necessary. 


Ice theory and methods, combined with specially designed 
adaptive testing strategies, can be useful in substantially 
reducing the number of items administered to’ trainees in 


_ar achievemént test battery composed of a number of specific 


contest domains (Research Report 77-6). 


° 


-Rotn Adaptive testing techniques and 1cC theory and methods 


are useful in reducing test lengths for tests used to make 
Mastery déqisions (Research Report 79-5). 

In a variety of applications to ‘the problem of achievement 
testing--ineluding’ measuring achievement with a large uni- 
dines ional item poal, measuring achievement levels ina 
dumber of specific content domaing,-and measuring achieve- 
ment against a ‘defined mastery criterion--adaptive testing 
rechniques using ICC theory can substantially reduce the 
numbers ot items required in an achievement test without 
reducing the quality of the measurements ,(Research Reports . 
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77-0 ahd 79-5). ae 


Adapt ive testing can improve the quality of achievement 
measurements in terms of both precision and validity while 
reducing .the aumbers of items ‘required \Research Reports 
7-72 and 78-4).~ 


ICC test’ scoring methods (Research Report 79-1) can be 
truitfully applied td achievement testing data (Research 
Report 79-3). However, maximum likelihood ICC scoring is 
less u#eful in conventional tests because of its non- 
convergence problem when the test is too easy or too s 
difficult for a testee. Although non-convergences occur 
much less frequently in adaptive test data, use of the 
three-parameter ICC model with different scoring methods 
tends to result; in’ somewhat, different achievement level 
estimates. More research on this problem is indicated. 
Because ut its ability to equate testings and link item 
pools onto a common metric, [CC theory has the potential 
of offering solutions to the problem of measuring gains” 
in achievement levels during the process of ‘instruction. 
However, examination of the dimensionality of an achieve- 
ment test item pool from pre-instruction to-the peak of 
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‘+ instructior shows changes in the dimensionality of achieve* 

‘mnt during instructton (Research Report 79-4). These results, 

it verified with other data, suggest potential problems in 

the applicability of unidimensional ICC theory to the measure- 6 
mene of individual: growth in achievement levels due to instruction. 


‘ 


bee The use of ICC.methods to measure retention following instruc- 
“tion was supported. by the data (Resear r¢h Report 79-4), These 
results .shay that the same achievement Variable was measured: 
up to ' month’ after instruction as was measured at the peak of 
instruction. : 

13.00 The? post-instyuct ion data (Res2arch Repore 79-4) also support 
the‘use of computefized. adaptive testing in operational in- 
structional envirdnmerits. Since.these data indicate that 
the same achievement variable is me&surable up to a month 
atter the end of’ instruction, instructional environments with. 
a Limited number of testing terminals can obtain similar 

‘, megsurements from trainees when tests are administered on 
; ditferent days. | ee \. 
le. The use of unidimensional ICC theory in achievement test ing 
is further ‘supported by the lack of effect.on dimensionality 
“or the administratiad of immediate knowledge of results 
Juring the process of achievement testing (Research Report : 
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Implications for Fufther Research 
tue rindings and experience of this 3-year research program strongly sup- 
woyt the use of fe theory and’ methods and computerized adaptive testing for 
the meisucement of achievement. However, many new questions were raiséd by 
the research (some of which were described above) and some of the original 
quvetions addressed are still in need of further research. Portions of the 
research deseribed below are being pursued under a contract entitled "Com- 
puceriaed Adaptive Achievement Testing,’ NR1I50-433, with the Personnel and 
traininy Research Programs of the Office of Naval” Research, with funds from 
toe Detense Advanced Research Projects Agency, Army Research Institute, Air 
Woree Oftdiee ot Medentifie Research, and the Oftice ot Naval Research. 
‘ 
Peter-subtest Branching - 
a 

Although Research Report 77-6 demonstrated that an adaptive testing 
strategy using in.ra-subtest adaptive item selection in conjunction with 
Inrer-subtest- adaptive branching could substantially reduce test battery 
length in one achievement test battery, the generality of this finding needs 
te be examined. In addition, the relative ef ficiéney of alternative approaches 
te inater-subtest branching needs to be studied. . : 

The scoring strategy used yin Research Report 77-6 was based on the maxi- 
min information item selection strategy using Bayesian scoring. However, the 
tae of Bavestan seoring, which has a tendency ta regress achievement estimates 
toward the mean, may result in the premature termination of the intra-subtest 
item selection, ee ly when used in conjunction with the minimum 
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information termination criterion. Thus, a relevant area of research is that 
of the evaluation of intra-subtest item selection strategiés that thay elinin- 
ate this problem and identification 6f situations under which use of Bayesian 


‘scoring in conjunction with maximum information item selection is less than 


opcimal. a 


. = 
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_. A second problem in intra-subtest adaptive item s@lection for inter- 
gubtest branching strategies is that of the termination criterion. Research 
to date has utilized a. termination criterion based on minimum information at 
the current estimated level of ackievement. However, if Bayesian scoring is 
to be used, it is possible to terminate on the basis of a minimum posterior 
Bayesian variance of the achievement level estimate. The relative performance 
of these two termination criteria as well as their interactions with the intra- 
subtest item selection strategy, needs to, be investigated. 


ry 


With regard to branching between contént areas, previous research has 
identified one means of ordering subtests for inter-subtest branching and has 


‘relied exclusively on linear multiple regression as the inter~subtest achieve- 


ment Level estimation technique. Other prediction strategies are available 
for making predictions between content areas and there are other, ways of or- 
dering subtests to be used in inter-subtest predictions. In addition, the use 
of Linear multiple regression equations brings up the question of sbrinkage 
with regard to the application of vegression equations based on one sample of 
individials when utilized on another sample from the same population. Tue 
effect of overestimation and shrinkage needs to be investigated within this 


-Inter-subtest branching strategy. 


- * 
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Finally, previous research has indicated that there: is wide variability\ 
in the range ot. reduction in number of items administered across subtests. 
Thus, a relevant question ig the nature of the subtests resulting in 

lurg&r or smaller reductions due to the-use of the inter-subtest branching 
strateyy. This latter question is most efficiently investigated by mon.ie carlo 
simulation studies in which characteristics of the subtests are systematically 
Varied, a 
Dimensionalits of Achievement Over Time ‘ 

As tndfeated above, ICC theory has the potential of permitting .the mea- 
surement of individual growth in achievement” over time in instruction. But the 
initial results in Research Report 79-4. suggest that the achievement dimension 
changes from pretest to end-of-course-unit testing. Thus, further examination 
ar this problem is indicated. 


The investigation of the dimensionality of achievement over time is being 


studied in a number of achievement domains,’ including domains that are primar- 


me 


ily cognitive as well as those that are primarily conceptual. Obtained data on 


‘achteveaent measured at various points in time will be factor analyzed. In 


each case, items will be parameterized by ICC models and the change of these 
parameters over time will be studied. In-addition, achievement level estimates 
hased on factors identified at celevant points in time will be obtained and the 
relationship among these achievement level estimates over time will be studied. 
The relative saliency of*factors identified at different points in time. will 
also be analyzed to determine whether the same factors are evident at differ- 
ent points in time but at different levels of saliency. If the latter 
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Foe whenle is supported by the data, it may then be poss.ble to investigate 
inter-time branching, taking into account the relevant saliency of mene’ dimen= 
sions at different soins in time. 
5 € Sad 

Depending oa the results of the analyses of achievement level data, at 
different points in time over a humber of instructional contexts, adaptive 
testing strategies for inter-time branching will be developed &nd° evaluated. 
If the same dimension is found to exdst: with different saliencies ‘at differ- 
ent points*in time, the utility of the information providea at tHe prior point 
in time with respect to addptive testing at later points in time will be 
studied by live testing and by real-data simulation, ‘One obvious approach 
would be’ to simply use the correlation‘of achievement level estimates on a. 
normative group from earl fer’ points in time with latdr points in time as ' 
entry points into Tater time achtevement level estimation. When data are 
avallable at more than one prior point in time, the use of multivariate pre- 


dicrion strategies becomes relevant, apd the relative advantages of different ° 


Stratesicvs will need to-be investigated. . \ 
‘ ‘ : , 
i ey 4 a 
Adaptive Mastery Testing ae : ( 
ee ee ey = . x a = ’ * . ¢ 


An adapts ve testing dtrategy for making mastery det isigns was developed in 
Research Report 79-5. Although the data in that report indicate some’ promise 
for this [UC-based mastery testing approach, considerable additional study of 
its potential as a solution to the mastery te/sting problem-is appropriate. \ 


First, the adaptive mastery testing (AMT) strategy needs to be studied 
in putational mastery tests, In addition, its operating chigracteristics need 
boone examined in corparison with competitive strategias fo yemastery eoEtne 


Lge fd boy strategies based on Waldian decision theory. \ : 
e ; ! ‘ nN 
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“43 Mier sirateyy also needs to be panied in a wide variety of classification 


sPQrit tens, in one application df the AMT strategy, error may be associated 
primar.ciy with the ertterion, as would We the case where the items in a mastery 
test are all‘of similar difficulty.an iscrimination; hence, the maximum in- 
Lormigion a the {tem pool is eer euapeee around the criterion cutoff value. 
lo. more rearistie situation, errors are associated with both the criterion 
and the dudividual being measured. These different ‘approaches: to adaptive 
Masters testing shouli be compared in both real-data- simulaFion studies and 
MONEE cays simulation &tudies. The real-data simulation studies will use ex- 
isting data administered ina gonvent ional test format, from mastery tests 
utfiieeasd tn military and educational environments, to determine the operating 
chara®teristics of these two major approaches to AMT as well _ ag to evaluate the 


outcomes when both the erfterfon and the individual are meastYed with error. 1%. 


ditferential results are obtained using these strategies in real-data simula- 
tion, it will then be appropriate to design monte carlo Simulation studies to 
model the relevant parameters of the situation (e.g., levels of item difficulty, 
discrimination, and numbers of .items’, as,well as various degrees of error on 

the eriterion) and to compare these, results with results obtained by: competing 
strategies. 


A ttualtarea of research with regard to AMT is the generalization oF the 
met he doloutes! to the mult i*subtest mastery testing -_problem. Similar to the 
sulti-subtest achievement testing problem, decisions made with regard to one 
subtest mav be related to decisions made with regard to another subtest. Thus, 
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research i's indicated for détefmining how much ‘information, derived fron one 
subtest in a multi-subtest mastery test can be used in adaptive testing in 
other subtesta. For example, achievement: Teysl estimates generated from sub- 
tegts in one content area can:be used to begin adaptive testing in another 
content area. In some casee, only very few items*will be necessary to make 
the mastery decisions id later subtests because:of the’ intercorrelations among 
the mastery decisf{ons and/or achievement level estimates derived from other 


. content areas. : : ; ’ 
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Aduptive Self-Reverenced Testing os a 4 
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The review ot the achfevement measurement jiterature indicated the lack 
of a coherent framework for the measurement of achievement. Approaches to the 
medsurement of achievement such as norin=y or population~referenced t sting and 
criterion-reterenced (mastery) testifig dppearedzto have nothing’ in common with 
each other and Llittletor no implication for what appear® to be the important 


problem in”the measurement of achievement -~hat of, measuring tndividual improve- 


, . ij 
ment in achievement levels during the process of, or as a result of, jastruc- 


tion, lhe attivities of the present research program have er se the notion 


of Adaptive Self-Referenced Testing (ASRT), which appears to r resent a cOo-. 
LOPent framework for the measurement of achievement. ASRI can incorporates 


into tainuele tramework the notions of inter-subtest branching, inter-time -¢ 
hranchit, and mistery testing. * ee 
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AT RE Us only possible by combining computerized adaptive testing and 1ce 
theery. It involyes the measurement of growth on an individual basis, incor- 
poratiug knowledge of the stydent's level of performance at an earligr point in 
tiae, which is used-as a starting point for measurement at a later point in 


time. SKYE is desined to track an individual's growth in one area of achieve~- 


ment as 1 function of time. Tt thus @an be used to identify the degree and 
extent det learning as it occurs and the point. at which learning occurs or fails 
2ovcur during the process of instruction. The generalization Of unidimen- 
sional sedt-referenced testing to the multidimensional case (i.e., where more 
thanoene content ared ig being measured) incorporates the intér-test branching 
prearlets Tie obivetive is to utilize, on an individual basis, information 
yicned both on other tests and at prior time periods for thet measurement of 
wreweh in dbearniog (achievement). 


. 


' 
Asal, is unigue in that the sequence of measurements taken to measure|\ each 


individual's learning history is based only on that individual's priagr pe! 
mapee at eartier peiuts in obéme in the same content domain. It is also des 
reo cperateeintiquedy within both computer-assisted «and computer -managed instruc- 
tien. Tt properly implemented, it should be an extremely Powerful approach for 
meisurios achievement in these contexts, permitting a cont inuous evaluation of 
Student progress and a. non-normative definition of "whené learning has occurred 
dad how mich has Been learned," while reducing testing hme to a minimum for 
earch student. . : / 
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be % Research Report 77-5 
i, PPPPTL? ap et an m Fool for the Adaptive Measurement of Achievement 
Isaac I. ea David J. Weiss, and G. Gage Kingsbury 
re September 1977 © 
the applicability of item characteristic curve (ICC) theory tc a multiplet 
‘ chuoiee test item pool-used to measure achievement is described. The 
: rationale for attemptimg to use ICC theory in an achievement framework is 
: summarized, and the adequacy for adaptive testing of a conventional class- 

room acttievement test item pool in a co 2ge biology class is studied. . Using 

eriterha usuall¥ ‘Wpplied to Ability measurement item pools, the it diffi-. 
‘ culties Nand discriminations in this’ ac‘ :evement test pool were found to be . 
simitar to those used in adaptive tes” ig pols for ability testing. Studies 
of the dimensionality of thé,pool indicate that it is primarily unidimen- 
sional. Analysis of the item parameters of items administered to two 
different samples reveals the possibtlity of a deviation from Yivariance in 
‘the discriminat lon parameter, but a high depred of invariance for the diffi- 
culty parameter. The pool as a whole, as well as two subpools, is judged to 
be adequate for use in adaptive testing. It is also concluded that the ICC 
model is not inappropriate for application to typical tollege classroom 
achievement tests similar to the one studied. , , 


‘ 


S , _ - Research Report 77-6 : 
te Saree OMettng Strategy for Achievement Teast Batteries 
: : Joel M, Brown and David J., Weiss 
° October 1977 , 


An adapcive teSting strategy nN described for use with achievement. tests 
that ener multiple content areas. -The testing strategy combines adaptive 
item chine both within and between.the subtests in the multiple-subtest 
hattery. A real-data simulation was conducted in order to compare the 
results from computerized adaptive testing with those from conventional . 
paper-and-pencil testing, in germs of test. information and test length. 
Data fora. the simulation consisted of test results for 365 fire-control ae 
a nicians on a paper-and-péncil administration of a 232-item achievement test, 
which was divided into }2 subtests, each covering a different content area. 
Correlations between subtest scores f rom adaptive and conventional testing 
were ,90 or higher for ‘ll, of “the 12 content: areas. An.information analysis 
showed that for all 12 subtests, the subtest information curves from adap- 
tive testing were essentially identical to the corresponding subtest infor- , 


items administered with adaptive testing was half as many as was required 
with convent tonal testing; ‘the shortest adaptive test battery used 18% of 
the total number of items in the convent tonal - ‘test, wnile the longest used - 
18. The adaptive testing strategy, therefore, pr pyided a considerable: re- 
dugtion in. test length and virtually ho loss in precision of measurement when 
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mation, curves From conventional testing. _ On the average, the number of ‘aero 


cdinpared with the conventional administratign of the achtevement test battery. - 
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- Redeareh Report 77-7 
An Invormation Comparison ef Conventional and Adaptive Teate 
In the Measurement of Crassroom Achtevement 
‘[gaac ti Bejar, David J. Weiss, and Kathleen A. Gialluca 
October 1977 
The information provided by typical and improved conventional classroom paper- 
and-pencil achievement tests is compared with the information provided by an 


adaptive test covering the same subject matter. Both tests were administered 


to over 709 students in a general biology course. Using the same scoring 
method, alaptive testing was found to yield substantially more precise esti- 
mates ot achievement level than the conventional test throughout the entire 


range of achievement, while at the same time reducing the length of the test. . 


The comparison of the improved conventional test with the stradaptive test 
also indicated that the-scoreg derived from the adaptive test weré more pre- 
cise, even in the range of achievement where the improved test was designed 

to be optimal. An analysis of the effects of, expanding an adaptive test item 
pool indicates that evens when slightly more dis¢riminating items are added‘ to 
the pool, improved precistjon of measurement can result. A ‘comparison of re- 
sponse pattern information values (observed information) with test information 
values (theoretical information) shows that! the ‘observed information consis— 
tently underestimates theoretical information, ‘although the pattern of: results 
from the two procedures is quite similar. It is concluded that the adap- * 
tive measurement of classroom achievement results in scores that are \less 


‘Likely to be confounded By errors p{ measurement and, therefore, are more 


likely to reflect a testee's true level of achievement. In addition, the re~ 


duction in rumber of test items administered by the adaptive measiirement of 


achievement can result in additional time spent in instructions ‘ 
| &j ‘ . 
; . 
Decadal: Report 78-4 , 


rote Kilt dartow of ddaptive Aehtavement Testing 

faant 1. Bejar and Ravid J. Weiss 

: November 1978 | 

Iie Sonabpitest validities of conventional classroom paper~and-pencil and adap~ . 
tive achievement tests were jcompared using data from'’two independent groups of” 


v 
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269 and °230 college students. Two adaptive achievement tests were computer ad-_ 


ministered to each group using the stradaptive: testing strategy; each group 
algo completed, two convent tonal classroom paper” and=pentil achievement tests. 
a achievement cests were drawn from the same ‘pool’of achievement test items 
on which item characteristic curve (REC), Parameters’ had been determined. 
Students were also administered two stradaptive'vocabulary tests. All tests 
were seored by maximum Likelihood estimation using the three-parameter logis« 
tic model. A nomological net was specified, describing the relationships of 
the achievement tests to the achlevement ‘constructs and their relationships 


with the tocabulary constryct and the vocabulary tests. The parameters of the. 


net were estimated by fitting the observed intercorrelations among the test 
scores to the nomological net, using the methodology of linear structural 
equations. Maximum likelihood estimates of the parameters cf the nomological 
net indicated essentially equal validities for the conventional and adaptive 
tests in four comparisons. However, the v validity of the adaptive tests was 


effectively higher than that of the conventional test's, since equal validities ~ 


were achieved with from 25% to 31% fewer items. The data also permitted an 
analyafis of the effects. of verbal ability on achievement test performance, 
separuitely for the conventional and adaptive tests. - The results from a con- 


firmatery maximum likelihood factor analysis showed a larger influence of 
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verbal-ability on achievement test performance at the first administration~of 
ss - the adaptive test.; This result was attributed to a nav-*sity to learn how to 
use the computer equipment with verbal instructions, which may have further 
reduced the validity of the adaptive tests. Combined with the facts that the 
adaptive tests were obtained under volunteer conditions while the conventional 
tests wdre obtained under "motivated" grading conditions, the results of this 
study indicate that computer-administered adaptive tests can provide more 
valid measurement of achievement than conventional classroom paper-and~pencil 
tests... 
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. Research Report /9-1 ° 

Sotuten besgnama Son Scoring Teste Data with Item Charaetertstte Curve Yodetls 

Isaac I. Béejar and David J. Weiss § 
ebruary 1979 _ 

Three computer programs are Yescri ed for scoring test response data using item 
characteristic curve (1CC), or latent trait, models. The rationale and math- 
ematical basis of both maximuth’likelihood and Bayesian ICC scoring methods are 
presented, as well-as some data comparing the two methods of scoring. The 
three computer programs ‘are designed for scoring conventional (linear) test 
data (LINDSCO) in aichotomous response format, adaptive test dichotomous data 
- (ADADSCO), and conventional (linear) test data scored by polychotomous ICC. 

si ; models (LINPSCU)+ Options ayailable in these three general purpose programs 

i . are deseribed, and examples of the input and output are given for each program. 
tompLlete FORTRAN Listings of the three programs ‘are included. 
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Research Report 79-3 _ 
ef one pe Among Achievement Level Estimates from Thrée 
trom cnaractertstte Curae feoring Methods 
,  . G. Gage Kingsbury and David J. Weiss 
’ April 1979 i 
this study compared achtevement level estimates from three item characteristic 
curve (LCC) scoring methods using the one-, two-, and three-parameter ICC 
madels. The three scoring methods were maximum likelihood normal, maximum 
likelihood Bb. sri¢, and Owen's (1975) Bayesian‘ scoring method. Data included 
‘all possible response patterns from a hypothetical five-item test, as well as 
response patterns fram Live admintstration of a conventional classroom and a 
computerized adaptive aghtevement test. For the conventional and adaptive 
test data, correlations among achievement level estimates were examined as a 
funetion of test length. Results for ail data sets showed a high. degree of 
similarity Among @ est {mates for the one- and two-parameter data, with.slight 
decreases in correlations as information on the discrimination parameter was 
“sed in seoring. When the third ("guessing'') parameter was used in scoring 
the item response data, correlations among 9 estimates were reduced, particu- 
larly for the adaptive test data. The data also showed an increasing tendency 
- tor the maximum likel Lhood methods to result in convergence failures as the 
Nhird parameter of the ICC was used in scoring. In general, however, the 
adaptive test data were less likely to result in convergence failures than 
were the conventional test data. The data also illustrated how each of the 
three scoring methods tend to utilize ICC parameter information in arriving at 
- estimates and the relationships of these estimates to a number correct scor-: 
ing philosophy. Advantages and disadvantages of each of the scoring methods 
are discussed, It is suggested that future research examine the relative 
validities of scoring methods and model combinations. 
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Research Report 192 
BPreet of Point in-Pime tn [hstruetion on the’ Measurement of Achtevement 
G. Gage Kingsbury and David J. Weiss a 
. August 1979 , 
Item characteristic curve (ICC) theory Kas potential, for solving some of the ‘ 
problems inherent in the pretest-test and test-posttest paradigms for measur - 
ing change” in achievement levels. However, if achievement tests given at 
different points in the course of instruction tup different achievement di- 
mensions, the use of ICC approaches and/or change scores from these tests is 
not desirable. This problem is fnvestigated in two studies designed to de- 
termine whether or not achievement tests administered at different times during 
a sequence of instruction actually measure the same achievement dimensions. To 
investigate possible changes in dimensionality between different points in in- 
struction, aspects of the dimensionality of achievement test data were examined 
prior to instruction, at the peak of instruction, and up to a month following 
the peak of instruction, Data used were conventional and adaptive achievement 
test data administered to students in a general biology course at the Univer- , 
sity of Minnesota. Results raised questions about the utility of the pretest- , 
test paradigm for measuring,change in achievement.levels, since a comparison 
ladieated that a change in the dimensionality of 

achtevement had occurred within the short (4-week) peletgd of instruction. This 


ww 


‘change was also observed using a factor analytic comparison. Use of the test- 


posttest paradigm to measure retention was supported, since a regression com- 

parison of students' achdevement Jevel estimates did not indicate any signifi- . 
cant change in the achievement metric up to 1 month after the peak of instruc- ~ 

tign. The signiticance of this result for the use of adaptive testing 

technology in measuring achievement is described. Implications of these 

studies and the use of ICC theory in the measurement df achievement, as well as 

rome potential Limitations in terms of generalizability of these results, are 

Jiscusssed. 
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Research Report 79-5 
eotisettte Teting clrateay fer Hastery Deets ions 
G. Gage Kingsbury and David J. Weiss 4 
September ‘1979 
‘ay an attempt ty increase the efficiency of mastery testing ‘while maintaingng 
a high level of contidence for each mastery decision, the theory and technolo- 
zy or item characteristic curve (ICC) response theory (Lord & Novick; 1968) 
and adaptive testing were applied to the problem of - judging inlividuals’ com- 
oeteneides avainst a prespec tiied mastery level to determine whether each indi- ‘ 
vidual is a "master" or a "nonmaster"” of a specified content domain.. Items “ 
trom two conventionally administered classroom mastery tests administered in a 
military training environment were calibrated -using the unidimensional three- , 
parameter logistie ICC model. Then, using response data originally obtained 
from the conventiondl administration of the tests, a computerized adaptive ” 
mastery testing (AMT) strategy was applied ina real-data simulation. The AMT 
procedure used ICC theory to transform the arbitrary "proportion correct” 
mastery level used in traditional mastery testing to the [C@ achievement metric 
in order to allow the adaptation of the test .to each trainee's achievement 
Level estimate, which was calculated after each item response. Adaptive test~ 
ing continued until the 95% Bayesian confidence interval around the trainee's 
achievement level estimate failed to contain the prespecified mastery level, 
At that point testing was terminated, and a mastery decision was made for the 
trainee. Results obtained from the AMT procedure were compared to results Jy 
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obtained from the traditional mastery testing paradigm in terms of the reduc~ 

‘ tion in mean test length, information characteristics, and the correspondence 
between decisions made by the two precedures for three different mastery levels 
and for each of the two tests. The AMT procedure reduced the average test e 
length 30% cto 81% over all circumstances examined (with modal test- length re- 
ductions of up to 92%) while reaching the same decision as the conventional 
‘procedure for 96% of the trainees. Additional advantages and possible ap- 
plicacions of AMT procedures in certain classroom situations are noted and 
discussed, and further research questions are suggested. 
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; : Research Report 80-1 

} BPN era Trmaliiate Knowledtye of Results on adil oveneal 

Soot Performance and Test Dimensionalfi ty 
: . Kachtecn A. Gialluca and ae J. Weiss , 
. January 1980 ; 

These two studies investigated the effects of administering immediate knowledge 
of results (KR) concerning the correctness or incorrectness of each item re~ 
sponse on a computerized adaptive test of Biology achievement. In the case of 
incorrect responses, Jhe correct answers were provided to the student. The 
results of these studies indicate that the ‘provision of informative KR did not 
systematically increase total test scores, as ,would be expected if students 
wert using informatiog from previously administered items to help them answer 
subsequent items. Furthermore, provision of informative KR did not alter the. 
dimensionalitw of the achlevement tests administered, indicating. that the 
latent trait. model assumption of local independence among the items was not 
atfected to anv significant degree. 


‘ 


4 


. 


re) 
ERIC 


~19~ . 


Other Papers and Publications 


Bejar, I. I. | A comparison of conventional and computer-based adaptive achieve- 
ment testing. In D. J. Weiss (Ed.), Proceedings of the 1977 computerized 


adaptive testing conference. Minneapolis: University of Minnesota, 


Department of Psychology, Psychometric Methods Program, 1978. 


‘i % 
‘Bejar, I. I.  Applicdtion of adaptive testing in measuring achievement and 
performance. In D. J. Weiss (Ed.), Applications of computerized adaptive 
testing (Research Report 77-1). Minneapolis: University of Minnesota 
Department of Psychology, Psychometric Methods Program, March 1977. 
(Paper presented at a symposium at the,18th annual convention of the 
Military’ Testing Association, October 1976.) . . 
’ : qo. : 
Kingsbury, G. G., & Weiss, D. J. Differences among three latent~trait-based 
- methods -for estimating achievement levels. Paper preserited at the 1978 


Convention of the American Psychological Association, Toronto, Ontario, 
Canada, August 1978. | , 


Weiss, D. J. Computerized adaptive testing for the measurement of achievement 


.and performance.. Paper presented at the conference on Innovations in In~ 
structional Systems Development, Wayzata, MN, August 1977. . 


Weiss, D, J. Computerized. adaptive achievement testing. In Harold F. O'Neil 
(Fd.),; Procedures for instructional systems development. New Yorks 


Academic Press, 1979. 


8 


Weiss,-D. J., & Brown, J. M. Multi-content adaptive measurement of achievement. 
inf, J. Weiss (Ed.), Proceedings of the 1977 computerized adaptive testing 
conference. -Minneapolis: University of Minnesota, Department of 
Psychology, Psychometric Methods Program, 1978. 


\ 


Oo 
G3 


. a 
’ : : : . 
' . % Py 
eey 1 Dr. awtiliaa Moonan : 1 Spectal Asst. for Education and 
Coue 32 és Training (OP-01£) 
havy Personnel haD center Rn, 2705 Arlington Annex 
Voir deck Ky. mersting San Ppega, JA 4e15 Seep Waghington, DC 20370 

Prove Rader teen: ad | . : 

Ve. Mew t Postage atu ile ahaa 1 LORK Ww. Moroney : lq, Office of the Chief of Naval Operations 
Martetegy, Ae a Co Je SOME ‘ Research, Development, and Studies branc 
om Naval Postyratuste Sehvol (OP-102) 

PT PEAYE TE be LB Ger. horberey, CAs dga0 Washington, DC 20350 

Sept. ot Natey ‘ f : , 

ra : 1, Commanting Cf ficer 1 Long-Range Manpower, Personnel, and 
; ere U.ty Navel Amphibious School ar Training Planning Brdnch (OP-110) 
tte Merar st plater * : Corunsde, CA 9155, Room Gbea 

eprint cf damina yee abiwe Coe ey Arlington Annex 
taees Pasr “ aba Tee ey % :° Library ; washington, DC 20350 

SEEM EY Gl OEE . : Naval Health Research Center - . 

, em rae PL ot. Row €S102 : 1 Nead, Manpower Training and keserves 
ee na ake ieee Sar Trego, CA 92438 section (Op-946'D) 

oe gs fires as : * Roon WWN78, The Pentagon F 4 

oe ! Ted M. of. Yellen Washington, DC 20350 
Technics! Information office, Code 201° 4 
Me Vile POE oa ‘ ~OMAVY PERSONNEL HAD CENTER 1 Captain Donald F. Parker, USN 
Liey Bet wrnee BAG) gee Gee * SAN DIEGO, CA 92152 << © Commanding Officer ° 
van Vheds, Aa tf : . oe : Navy Personnel R&D Center . 
; 1 Library, Code PeoIL fan Diego, CA 92152 
Poo. denn Fors Navy Personnel R4&D Center _ 

avy Persurned ay lether ge! sam Dieqo, CA 92152 1 LT Frank C, Petho, NSC, USN (Pq.D). 
Tar Phemge. od ade? ; : : , se Code L51 ’ : 

; » Technical Director Naval Aerospace Medical Research Laborat 
Ve fe, Patrscw Koo thao Navy Personne! R&D Center Pensacola, FL 32508 

Pays polomy cvurse Carector  - oan Diego, CA 92152 : 
LEADERSHTP & Law SEPT. (tp) 1 «DR. yRICHARD A. POLLAK 
NIV, oF PRORESOEINALS CE VE LOPMMERT 1 Director, Navy Personnel R&D Center pu Se COMPUTING CENTER 
‘a NAVAL AGADEMY dashingtom Liason Office U.S. NAVAL ACADEMY 
AXNAPULLA, MD gna ,  Purlding 200, au ANNAPOLIS, MD 21402 . 

Wushington Navy Yard, DC 20374 4 
! Or, Norman J. Kerr ? z . : 1 Dr. Gary Poock , 
Jthtefoot Naval Teannical Troasming 2 + Commanding Officer = Operations Research Department 
“Naval Ate “tation Mempnys (18) Naval Research Laboratory Code 55PK 
Malbingtor, IN 3054 | . Code 2627 * Naval Postgraduate School 
, , Washington, AC 2nign 7 7 : Monterey, CA 93940 
' Shy heard Krowcer . ' - : , ° . ; 
Navy Personeel Kati lenter “4 Psychologist : 1 The Principal Deputy Assistant 
ean Phemwo, CA 1 14e ‘ ONR Branch Office Secretary of the Navy (MRA&L) 
,; os ee an D 4E780, The Pentagon 

or. Willtam wu. Malay = 266 Summer Stree 3 n 

Principal Civiitan Advisor for Boston, MA 92210 , pastiingtom, De oo . 
Education and Treqining 1 Dr. Richard Sorensen 
Navel Trareiag Command, Code 234 1° Psychologist Navy Personnel R&D Center 
Pensaceia, FL 2698 ONR Branch Office  ° San Diego, CA-92152 
; 56S. Clark Street — ‘ 
: 3 5 : 4 Weitzman, 
! Or, Kreaie derspals SHAS aK Ey ee, BuO oobe saa are 
ae Alvigor ta DoM.cMpT) . ¥ Drie OF Naval Redearen Department of Administrative Sciences 
; S faknk sii Ae se Code 437 U. S, Naval Postgraduate School 

Washington oC ect : 80 N, Quincy SStreet Monterey, CA 23940 + 
“Ayr Repay | Seage Ba: - Arlington, VA 222717 : 

! and. ue ils ere - : . a ; 1 OR, HLM, WEST ILI (OP-01) 

Prospective vamnmandiag ctficer - 
vo Cari @s6 son (CVNet) a > pelle & oo Research Programs eee iba 
Newoort News Shipburiding and Drydec« Co te 4 : 5 
Wewpurt News, VA oats mw Office of Naval Research WASITNUTON. 0G -20,550 , 

: Arlington, VA 22217 } 
‘ ‘ 1 Dr. Robert Wisher 
! Pr, James MeBrage 4 p ial t Code 309 -_ 
ss al ty OB ap a Ss . i 
per aia ee ONR Branch Office : Navy rersonnel, Bab Senter 
AF os A pete . iN ‘ 
: 1930 East Green Street Son DiegOy. Gk. Je toe ; 
To fr. George Moed ber + Pasadena, CA 91101 


1 DR. MARTIN F, WISKOFF 
NAVY PERSONNEL R& D CENTER 


deat, Human Factors Uepe, 


Raval Submarine Metical Research Lab ! Setentific Director 


Sroten, TN Cg - office of Naval Research SAN DIEGO, CA 92152 
. “Selentific Liaison Group/ Tokyo 1 Director, Research & Analysis Division 
Toe wt alia Mor ague American Embassy Plans and Policy Department 
Stey Persunnet RAD Ceeter APO San Francisco, CA 96593 Navy Recruiting Command 
wer Dpege, TA W2tho e 4015 Wiisoen Boulevard 


Arlington » VA 22203 


ERIC : 34 


Qin 


1. 
* 
9 «4 
2. 
ry € 
a 
@ 
. 
Yo 
ty ( 


o_. 
ERIC 


1 
: 4 
a” 
c 
ur jernord kiaiant CTR) . 
haoe Papsonael 44D lenter 
ee Shed oy TR tre 
’ awe ‘ 
: fob Nay ri thas emi Trainin 
Nat 4 
‘ rorep Sa ft 
‘ a wer a 
ffs oe ot ohaedt GP ‘ay Cperat lors 
} earl 
eeu A ee 
He ; 
a Notre ¢ & 4, ite 
' PO 2 hotve. et? bon uroup 
Tar. 
oan Ede Ney 
range, PL an 
4 
ar ay 
Saeki Carector : ‘ 


s 


° 


VAP yy wee are 


Sesoows orphan amd Aig. olentes 
Uae tse mee SRE . 
ALegetie ad, YR stage ‘ . 
7 


Mol. Barbee 
Hat, Separtinent of 
GAPE «tbe 

do htreytton, OV 


tie ary 


vy 


Coa ary a, BL owefore 
"Saray TRACE Cyskemy 
ALtm: ATAAsCa 
wok, Natty \ 
oh, RALPH TEKS, 
“yo, ARMY YES SARSM DNS TITTe . 
YORTAENa WER ANF NGF 
ALEXANDRIA, VA dea bat 
tat 


* 


Se Myrout Fisehl 
Wea. Army Reser 
Wek at Met ivloral 
‘SS kigenreseer Ay thie 
Rardardria, VA Le i? 
Ur. FA Jomnsos 
Army Xegearlt institute 
Vth Sasenrower Biv d, 
Aveaandraidg. fA Oe 


mr. Myctooel Kapian 
ARMY RFOEAROH [45 
te OST DENA WER AVENUE 


NLEXANROR EA, VA Gaga 


STlryte 


Or, Mibron Katz 
training Teommacal area 
Army Reseach Tastitute 


Eisenhower Ay enue 


aA 


Alexwanmdrua, VA P2532 
or, Hareat Boo. Mea Sh, 
Aeein: PER TA0K 


Army Research institute 
SOC) Fiserrower Avenue 
Alexa drial VA ets 


‘Statute for 


in " 


we Lences 


x 


Mmotitute for the 


t 


Aner yS18 Activity 


1 


ils: 4, 


‘Life Sciences Directorate, 


d 
LT? Michael Plummer 
Cnief, Leadership & Orguniaationnl 


“3 


frfectiveness Division 
Tice of the Deputy Covet a0 
fur Personnel 
Depr, Of che Amay 
Pent spon, Washington pe 401 


OR, JANES Le REYEX aa. 
U0. ARMY RESEARCH INSTITUTE 
G91 EYSENHOWER AVENUE | 
ALEXANDRIA, VAQOS 322 


Mr, 


te 
heat 


Robert koss “ 
fs. Eysenhower Avenue 
Aloxaniria, VA Pete 


fr 2 Robert. 
u. os, 


Sa smor ' 


Lehavioral and Socer.s) 
S801 Fisenhower Avenue 
Alexaniria, VA 222337 
Commandant 
gS ag Institute 
« Sherri) 


’ 


stuff 


. Army Research Institute for the 
Tecial and benavidrur Sciences 


Army Research Institute for the 
Se1ences 


of Administration 


Attn; 
FT St Beer Harrison, I 46054 


Or. Frederick Steinheiser 
Army Reserch Instituty 
5001 Eisenhower Avenue 
Alexandria, VA 22233 


; Dr, Joseph Warepte 


U.S. Army Reseapehn Institute 
5901 Eisenhower "Avenue 


Alexandria, VA (P2333 


x 


Air Force 


Air Force tluman Resources Lab 


AFHRL/MPD 

Brooks AFI}, TX 78245 

U.S. 
Research 

NL 

Bolling Air Force Base 

dashingtén, DC 20332 


Dr. Furi A. Alluisi 


HO, AFHKL (AFSC) 


Brooks AFB, TX 782% 


Dr, Philip De Leo 
AFHRL/TT 
Lowry AFB, CO Neto 


Dr. Genevieve Ha wdiad 
Program Manayer 

Life Sciences Directorate 
AFOSR . 


sBolling AFB, NC 20240 


Air Force Office of Sqientifiec 


Research and Measurment [Nvision 


Research Brunch, AFMPC/iMPCYPR 
Randolph AFB, TX 7al4a = ° 
far 


1 


7 HO, US, 


Dr. Malcolr hee 
AF''SL/NP a 
Brooks AFB, TX 78235 


Dr. Marby Rockway CABHREZT 


Lowry AFB Y ° 
cone $02 tn 

: e 
Dr. Frank Schufletowsk1 
U.5. Air Force 
ATC/XPTD os 


Randulph AFB, TX, 78145” ~ 


Jack A. Thorpe, Maj, USAF 


Naval Var College 


Providence, RI 0254ufq 
rian K, Waters, Lt .Col, USAF 
Ajr Mar College (EDV) 
Maxweli AFB, AL 26112 
4 
Marines a 
a 
’ 


v 
He William Greenup ‘ 
Educatioh Advisor (E031) 
Education Center, MCDEC 
Quantico, VA 22134, 


ae ee Office of Manpower 
HQ, Marine Corps (MPU) 
PCB, Pldg. 2009 : 
Quantico, NA 22134 


Headquarters, Us S, Harap ie 
Cade MPI-20\7 & 


Washington, petlen 380" 


’ 


Spectal Assistant for Marine 


Corps Matters 
Code 100M 
Office of Naval Research 
890 N. Quincy St. : 
Arlington, VA 22Q17 . 


Major Michael L. Patrow, \SNC 


Headquarters, Marine Corps 


. (Code MPI-20) 


Washington, DC 20480 


DR. A ., SLAFKOSKY > 


t 


Utilization 


torps al 


SCIENTIFIC ADVISOR (CODE RD-1) 


MARINE CORPS ° 
WASHINGTON, DC 20380 


& 


CoastGuard ' 
Mr, Thomas A, Warp | 
U, S. Coast Guard’ Institute 


P, 0, Substation 18 
Mlahoma City, OK 73169 


19 


washington, OC 


“ry ok wale ase 


Jamgtihaae oan 


“Safe far 


afecrsue © Sar etoph sor amen 

eter CP hand, Plae. 
Alegatdeia, YA Ino Sa 

sy? Ma ‘ 
Aeters | . y 
Se Spgt tee bye or ce 
; a a. 
SEVAMR ED at baa ob ba CELT AGEN Y 
fads ow hoc ell, * . 
AML ENG ON, VA Lh 
s 

te Vs PP wt, 

Py. omliibare or 
PPSh amg Cyteet te 
Pesan BO te Be woe w 4 
Be Cher ebony The 

’ ara 
Lpreet | Keser bh ord Mata 

SAD ONE AGL 9 ‘ 


orson, he Peers, ido. 

Hashing tar, aa 

Mysutary As ststant. for ahd at 
Pere ta Tenn. Wey 

eTbbeor of She om fer 
far ge searer. rAgingerityg 

Rowe Bod, Tha Pootagar v3. 


* 


agamningtaum, Mo west : \ 


Trial tecg, 


a : 
MAT OR oc tytee Sey ditan, JAk 


Mths at the Aygistaun. wore ary, 

defense VRARLS o 
Ma Tere Pears un ae 
masteamg ror, Me ots 


o. . 


Tivad Gow 


Ur. Lorraine be Fyde \ 
Personnes; af enter s 
Hhiee of Personnel Management Of USA 
‘He Ptr} NW 
arsbattarm, l.  ay 
1 
gerry Lennus 
REGLONAL PoYoOeLOG tot 


Hoy, ef Pbae of Pergoones Management 
cae, DRARHO RN OTREET « 
THES AGE UT ntti va 
i 
wthbiwog. “etauron 
Rm 9) interna Revetue Service 


20) lefferson Savis Wigt «ty 
Arhangter, WA 22s. 

‘ . 
“rp. ANirew #, Moleur . 
TYP ne Ryu am yan dev. 

ant &egearch . 
National Se tease Fourdat ban 
mashragte oly Ora 


Personne, Wo Center , 
fT fhave af Pergsconet Managnekt 

49°. F Brreet NW 

og Phy 


SYMaLco 
Program pire tor 
Manperwer Regeqreb otf Ade rae ry Cwrvices 
Iecsty tart pase 

AW Noreen Pitre utereet 


Avex amdrya, sA Mita 


“ere tiny asf Oe er se 


Dr .SVern mw. Urry 
Personne! &&N Center 
Cefaice of Personnel Management ~ 
108 E. Street NW 

dagnington, DE 20415 s 


Dr. Josepm bL. Young, Director 
Nemury & Cognitive. Processes : 


National Serence Fuundaution aie 


wastingtur, Pe 20559 


Hon ‘Govt 


Dr, FPrjing #. Anttersen 
Department af Statistics 
Stud lestravde & 

1455 Copennagen . 
DERM ARK 


, 
1 psychological research unit 
Dept. of Defense (Army Office) 
Campbell Park Offices 
Canberra ACT 2@j10, Australia 


Ms, Carole A. Bagley, 
Minnesota FAaucational Computing 
Consort lum 
m4 Hidden Valley Lane 
Stillwater, MN 55082 
to 


Dr. Jackson Reatty- - 
Department of Psychology 
University of California + 
Los Angeles, CA 99024 

Dr. Isaac Be yar 2 7 
Educational Testing ‘Service 
Princeton, NJ 8450 


De zWPs im Streitkraefteant 
Postfach: 20 30 03 2 

D-530N Ronn 2 i: «3 
WEST GERMANY 


re , Nicholas a: Bond 

Pept. of Psychology 

Sacramanto State College 

ON Jay Street 7 é «at 
Bacramento, CA 95819 =e 


Or. Lyle Pourne, 
Department of Psychology : 
University ef Colorado 
Raider, CO A309 

= 
Mr. Robert ee s 
American College Testing: Programs 
P., Cy Box 163 
Towa City, TA 52249 


DR. G. VICTOR BUNDERSUN 
WICAT INC. 

UNIVERSITY PLAZA, 
1160 S50. STATE ST. 


OREM, UT @4N57 . - 


SUITE 10 


Or, John B. Carrol} - 
Psychometric Lad 

Univ. of No. Carolina 

Davie Hall 1A 


Cnapel Hili, NC 2TH VA 
26 
a 
« * *. 


b 


1 


1 


1 


1 


Charles Myers Library 
bLivingstone House 7 
sLivingstone Ro: ‘ 
' Stratford 
London E'S 2uu a 
ENGLAND ° «. ' 


Dr. Kenneth E. Clark 
College of Arts ‘S Sciences 
‘ Un¢versity of Rochester — 
River Campus Station 
Rochgster ,_ NY 14627 


Dr. Norman Clrrt : . 


’ Pept.-of Psychology 
Univ. of. So. Califormia 
_Unaversity Park ‘ 
Los Angeles, CA 99097 
$ 
Dr. William &£. Coffinan 
Dirgctor, Towa Testing Pragrans 
_ 334 Lindquist Center 
University of Iowa 
Towa Citys IA $2242 


Dr. Meredith-P, Cruwford@ 
Ameriéan Psychological Association 
- 1200 17th Street, NW, ; 
‘dashington, DC, 20036 Me 


“De. Kenneth P, Cross P 
Anacapa SCiences, Inc, 
P.O, Drawer Q 

Santa Barbara, CA 93102 


ERIC Facility-Acquisitions 
483% Rugby Avenue 
Bethesda, HD 20014 


Dr. Legnard Feldt 

Lindquist Center for Measurment 
University of Iowa — 
IowavCity, IA 522h2: 


Dr. Richard L. Ferguson ° 
Tne American College Testing Program 
P.O. Box 168 ; . 

Iowa City, IA 52240 - 


Dr. Victor Fields 
Dept. of Psychology 
Montgomery College 
Rockville, Mb 70850 


- Univ. Prof, Dr. Gerhard Tischer 


Liebiggasse 5/3 7 
A 1010 Vienna 
AUSTRIA 


Professor Donald Fitzgeraid 
University of New England . 
Armidaie, New South Wales 2351 

, AUSTRALIA ‘ 

Sas 
Ww. Eqwin A. Fleishman 
Advanced ReSearch Resources Organ. 
Suite 900 
4230 East West Highway 

- Washington, DC 20014 


Dr. John R. Frederiksen 
Bolt Beranek & Newman 
50 Moulton Street 
Cambridge, MA 02128 


“ 


et 


~ 


aaSx: 


1 


ray 


ra) 
-ERIC 


1 


7 


: 


“URIVERSETY OF PITTURUAGH 


CH. JANE, 


. 
Or, bari Muay 


x 


OR, ROBERT GLASER 


LROC 


Ee HARA 


: PUTTS ‘BUAGH, 


t 
Russ 


Or, 


OTREET 


PA 15212 


areen ¢ 


CUB Me Graw OTL 


Led Hout + 


Tanterey, CA 


Lars 

INEVE@ ITY 
DARA 
Pre tonbaad, 


- 


Ay 's 


he gears. 
s 


eo hy 


P pitts 
UTRERT 


PAS 


Poth 


GREEN: 


ce an 


ALU Med 


tat. Ra: bombirtan 


eoboGd 


at 


Pducety ? 


‘Ieaver sity 


Aber st, 


Ma 


rh oF tants 


Vag 


rr, 


A aeatter fourrg % 


Department 


eeturd al Educuthern 
Mriversity of Gaitforns.s 1 
Sabla Barbora, Cs ast 


Ur, Paederbon Mayrredoth 
The Rand Corporation 
10 Gain at Paved, 
Sante Monto, Cf rue 
Qe, Lhoyd Mamphre ys 

uf Psygpolozy 
University of ftliauis 
Crapalat ye IL eit) 

3 , 

Libro y 

WUARR. feestorn Pryanton 


aan? Herwiek Orive 


armel. CA 
‘ : 

De, Steven Hunka- 

Seger tment of Edu ition 


oe ian | 


aieersyty of Alber’ + 
Fimentar, Yebert, 
LAMAQA 


Dept. of Puychobes« 
Unbeersity of wmashingt or | 


wptle, we tps tts 
oer, tuynts Magn . 
Colvege ot Edue ation 


Universety of Sounh Carobtind 
ee ed 


Tathumtag, Tf 23 
Are. Dosages BY Janes 

fim Taye 

fy ucational Testing Service 
Princeton, No Uaege 


* 


dournal Cuppiement Aetract Service 
American Psyctutogi: i sanc ation 


Lo F*tnh Street NW, 
washington, AC gle \ 
Dr. Araold FL Kanarieck 
Honeywoil, Inc. 
Seoneyweli Plaza BNI STFA 
“inneapoiis, MN 58499 

“ 2 NW 

n° 
4 / 
= 8 


6 
Professor John A, Keats. 
Untversity of Newcastle 
AUSTRALIA 2308 ; 


‘ my a 
Dr, Mazie Knerr ¢ 
Litton-Mellanics : : 
Box. 12°6 
oprangfield, VA 22151 ‘ 
bd e 


Dr. Stephen Kosslyn ~ 
Harvard University 
Department of Psychology 
3% Kirkland Street 


Cambridge, HA 02148 : ‘ 


Or. Alan Lesgold 
Learning f&b Center 
University of Pittsburgh 
Pittsburgh, PA 15260.” 


for. 
210 Education Build ity gat 
University of Tllinois 
Champitign, If 61820 

Pr, Cnarles seni 7 
Fuculteit Sociale Wetenschappen 
Ri jksunaversiteit Groningen 
Oude foter ingestraat 

Gronismen 

ReTHERLANDS 


s 


Or. Ruban’ Linn 
College of Education 
University of Illinois 
Urbana, TL 61801 


Dr. Frederick M. Lord 
Educational Testing Service 
Princeton, NJ O85u0 = 


Dr. James Lumsden 

Department of Psychology “ 
University of Western Australia 
hedlands W.A, 6909 ‘Ne 
AUSTRALIA 


Dr. Gary Marco 
Educational Testing Service 
Princeton, NJ 08450 


Pr. Scott Maxwell r 
Department of Psychology 
University of Hodston 
Houston, TX 77004 


Or, Samuel T. Mayo 

Loyola University of Chicugo 
420 North Michigan Avenue 
Chicugo, IL 60611 


Or. Mark Miller. 

Computer Seclence Laboratory 

Texas Instruments, Inc. 

Mail Station 371, P.0. Box 225936 
Nallas, TX 75265 


Professor Jason Millman 
Department of Education 
Stone Hall 

Cornell University 
Ithaca, NY 1995 3 


€ mye 


z 
Michael Levine a pe 


1 


1 


| 


1 


wv 


Or. Melvin R, Novick: 


396 Lindquist Center for Neasurment 


University of Jowa 


Towa City, IA 522h2 ‘ 2 


Dr, Jesse Or lansky 

Institute for Nefense Annlyses 
4n0 Army Havy Prive . 
Aryington, VA 22202 — 


»Dr. Jamés A. Paulson ‘ 
Portland State University 
2 P.O. Rox 751 
Portland, or Y97207 


“4R. LUIGI PETRULLO , : 
‘231 N,,EDGEWOOD STREET 


ARLINGTON, 


VA 22207 


DR. DIANE M, RAN SE "YoKLER 


Mek RESEARCH 4° 


SYSTEM DESIGN 


39&7 RIDGEMONT DRIVE 

MALIBU, CA 90265 ‘ 
MINRAT 4t, L. “RAUCH 

P.Il 4 


BUNDESMINISTERIUM DER “VERTEIDIGUNG 
POSTFACH 1328.—., 
D=-53 BONN 1, GERMANY = : 


Dr. Mark PD. Reekase * ' 
Educational Psychology Dept. 
University of Missour t-Columbia 
4 Hill Hall 

Colunbia, MQ. 65211 

Pr. Andrew M. Rose ; 
American Institutes for Research 
1055 Thomas Jeffersoh St. NW 
Washington, NC 20007 

Dr. Leonard L. Rosenbaum, Chairman 
Department of Psychology 
Montgomery College 

Rockville, MD 20850 « 


Dr. Ernst 2, Rothkopf 
Bell* Laboratories 

600 Mountain Avenue 
Murray Hill, Nv 07974 


Dr. Lawrence Rudner 
WO’ Elm Avenue 
Takoma Park, MD 20012 


- 


Dr. J. Ryan 

Department of. Education 
University of South Carolina 
Columbia, SC 29208 ‘ 


PROF, FUMIKO SAMEJIMA 
DEPT. OF PSYCHOLOGY 
UNIVERSITY OF TENNESSEE 
KNOXVILLE, TN 37916 


DR, ROBERT J. SEIDEL 

INSTRUCTIONAL TECHNOLOGY GROUP 
HUMRRO 

300 N. WASHINGTON ST, 

ALEXANDRIA, VA 22314 


3 


\ 


. ~ : . : : : i 
2 s vs. 


meh vu Ln semen = decree nee eet aenteneemnen ceccmeeeee eee ee ee eeeennntetetann cnt renncnean sateen an aes Stee Serums ap 89 geen Bema z 


e 2 
e * 
t Or, Keadus Shigemasgu . 1 Dr. J. Unlanee E . . : _ 
University of Tohoku Perceptronics, [nes - = ‘ 
Department of Educational Psychology 6271 Variel Avenue ; im a a ‘ 7 
¢ Kawauehh, Seafat 949 . Woodland Hills, CA 91364 ~~). %, : 4 ; 
JAPAN ay ; ‘ e 4 , 
. . e 1 Dr. Howard Wainer ; 
b, fr. Pawin Snookey Bureau of Social SCience Research F : an. 
“Department of Pay talgay 1990 M Street, NeW, ‘ = ig 
‘ » tiniversity of Central Florida ' . Washington, DC 2n03f Pe 3 ; : 
. Orbanto, FL VB, ; 
F 1 Or. John Wanneus * 
JY Dr. Haters. Senath . Department of Management ; ba 
é DO partaunt of Computers Seatenge Michigan University . : * , 
Rutpers University ‘ East Lansing» MI, WBR2N 7 : 
Hew Krunswick, NJ Gad? . 4 : tw : : > 
, 1 Dr. Pnyllis weaver A. , 
1 ocOr. Richart vnow ' : Graduate School of Education 
Stool of Efueation, . Narvard University ‘ 
Stanford Uniyer sity 200 Larsen Hald, Appian Way 3 ow 
Stanford, CA qaree Cambridge, MA 02138 
1 Dr. Kathryn T. ‘ipoenr = ¢ 1 DR. SUSAN E, WHITELY ‘ ; : 
Departiént of sycholony PSYCHOLOGY DEPARTMENT ; % - . * 
Brown Universi tye UNIVERSITY OF KANSAS Los . : . 
Providence, RE Oddly JLAWRENCE, KANSAS, 66044 ; : . . ry 
4 : : . Oe : : 
ae Or, Robert. Sternberg “ 1 Wolfgang Wildgrube * . s: 
Dept. of Pa yore logy * Streitkraefteamt a 
Yale iniversity ‘ ' Box 20,50. 03 
Box 11A, Yale Station ¢. _* De5300 Bonn 2 
Hew Haven, CT  OASO9 WEST GERMANY 
1 oD, avid Stone : . : => 
) FD ah ' a ; s 
: SUNY, Albany : . - 
Albany, NY toe? . z : 7 
’ . . : : f 
1 OAR, PATHE CK SUPPES Se f . ‘ 
PITS Suits, Bok MATHRMATICAL CTEDTES tu a ; an 
THe SECTAL G2PRNCES 7 A 
STANFORD UNTVE RS ITY ’ 8 : 
STANF ORE. CA Hadas : . “ . Se, ; 
+ 
1 Or, Maer tna an Sw aie 2 va : 
Laboratory ‘f Paycnometris ana \ : ; 
Fyaluatiun Researer hy ve : : ‘ 
Sehoe toot Fidue ation 
Untverarky af Massachusetts 7 . oe = ; 
Amberst, MA Me : : ’ , - 
! Dr. Brad Oyapson » 7 
Pay. hometr, > “Rear ey iwsuap . ar 
‘ Educ ge sonat Testing Service . ™ « 
p Princeton, Sy Seat ~*~ : 
: % : \ re 
1 or, Kiaumi Tat suoKa ) ; { . 
Jomputer faded Eduration kesearch ' 
Laboratory ° 
Sod Figine tank fdeseaest Laboratery .” . , * » 
Untveravy of Th linois . A 
Urbana, IL fig > . 
' Dr, Mawrd Thisaen : ; « 
+ Department of Paysho logy . ' v 
\ University of Kansas - 7 
Lawrened, K3 f’60Kg 4 
. een . ¢ « 
! Dr. Nouqids Towne . = 
Univ, of So, Taitforpta ; a ' 4 : 


Behavioral Technvloagy libs 
TT rays oN. Ebena Ave, $ 


edorta Reaet ys CA yt? : f) = ; 
? » y 


> \ os = re 
ERIC | ; | <. 
rs a . : . “ : : SSydf i 


, PREVfouS PUBLICATIONS 


. | a tae ge} . 
Proceedings of the 17? Computerized Adaptive Testing Conference, I a 


. Research Reports ' 
: baad 

Effects of Immediate Knowledge of Results. on Achievement Test Performance’ 

and Test CTimensionality. January 198¢. ig : 

The. Person Response Curve: Fit of Individuals to Item Characteristic Gurve 

Models. Lecember 197%, . . ‘ 

Efficiency of an Adaptive Inter-Suttest Branching Strategy in the Measurement 

oof Classroom Achievement. November 1979. +. * 

_ An Adaptive Testing Strategy for Mastery Decisions. September 1979: 

Effect of Point=-in-Time in Instructico on the Measurement of Achievement. 

August 1679, + ORY 

Relationships among Achievemeht Level Estimates frum Three Item Characteristic > 4 

Curve Scoring Methods. April 1979. : sl 

Final Report: EBias-Free.Cofiputerized Testing. March 1979, 

effects of Computerized Adaptive Testing on Black ang White Students. March 

127%, . , a 

Computer Programs for Scordng Test Data with Item Characteristic Curve Models. 

February 17S, | 

An Item Elias Investigation of a Standardized Aptitude Test. December 1978. ’ 

A tonstruet Validation of Adaptive Achievement TestingS November 1978, 

A Comparison of Levels and Dimensions of Performance in Black and White Groups . 

Cn Tests of Vocabulary,.Mathemat{és, and Boatie) Ability. October 1978, — 

The Effects of Knowledge of Résults and Test D4 ficulty on Ability Test 

‘Performance and Psychological Reactions to Testing. September 1078. 

A Comparison of the Fairness of Adaptive and Conventional Testing Strategies. - 

August 1278. . . pao . 

Arn Information Cemparison of Conventional and Adaptive .test’s/in the Méasure- 

ment of Classroom Achievement. October 1977, V 

An Adaptive Testing Strategy for AChievement Test Batteries. October 1977. 

Calybration of an Iter Pool for the Adaptive Measurement of Achievement. 

‘September 1977.. “ 

A kapid Iteg-Search Procedure for Eayesiaw Adaptive Testing. May 1977. 

Accuracy of Perceived Test-Item Difficulties. May 1977, = 

A Couparisoan of Information Functions of Multiple-Choice and Free-Response 

Vocabulary Items. April 177. - sy 

Applications of Computerized Adaptive Testing, March 19977, . 

tinal Report: Computerized Ability Testing, PS72-1975. April 1876. 4» 

Effects of Item Coaracteristics on Test Fairness. Beember 197€,;° 

Psyehclcgical Effects of Immediate Knowledge of Results and. Adaptive Ability 

Te-tinge. June 1976. ; 

Ef-ects of Immeaiate Knowledge of Results .and Adaptive Testing on Ability 

Test: Pertormgatce. June 1976. ae ai 

Effects of Time Limits on Test-Taking Behavior. Avril 1976.” ! 

Sone Properties of a Bayesian Adaptive Ability Testing Strategy. March 1976, 

A Simulation Study of Stradaptive Ability Testing. “December 1975. " 

Cogputerized Adaptive Trait MeaSurenent: Problems and Prospedgs. November 
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A Study of Computer-Administered Stradaptive Ability Test4ng. Octoder 1975. 

Empirical and Simulation Studies of Flexilevel Ability Testing. July 1975. 
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March 1¢7&, . 7 . 4 

An impiricsl Comparison of TworStage and Pyramidal Adaptive Ability Tésting. 
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Sirulation Studies of Two-Stage Ability Testing. \October 1974, 
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A word Khowledge Item Pool for Adaptive. Ability Measurement. June 1974. 
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Comparison of Four Empirical Item Scoring Procedures. August 1973, 
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